In the process of driving， autonomous vehicles need to complete target detection， instance segmentation and target tracking for pedestrians and vehicles at the same time. An environment perception model was proposed based on deep learning for multi-task learning of these three tasks simultaneously. Firstly， spatio-temporal features were extracted from continuous frame images by Convolutional Neural Network （CNN）. Then， the spatio-temporal features were decoupled and refused by attention mechanism， and differential selection of spatio-temporal features was achieved by making full use of the correlation between tasks. Finally， in order to balance the learning rates between different tasks， the model was trained by dynamic weighted average method. The proposed model was validated on KITTI dataset， and the experimental results show that the F1 score is increased by 0.6 percentage points in target detection compared with CenterTrack model， the Multiple Object Tracking Accuracy （MOTA） is increased by 0.7 percentage points in target tracking compared with TraDeS（Track to Detect and Segment） model， and the \mathrm{A} {\mathrm{P}}_{\mathrm{50}} and \mathrm{A} {\mathrm{P}}_{\mathrm{75}} are increased by 7.4 and 3.9 percentage points respectively in instance segmentation compared with SOLOv2 （Segmenting Objects by LOcations version 2） model.